Pii: S0893-6080(97)00032-4

Analgorithm for addition and deletion (ADDEL) of resources during learning is developed to achieve two goals: (1) tofind feed-forward multilayer networks that are as small as possible, (2) tojind an appropriate structure for such small networks. These goals are accomplished by operating alternately between an adding phuse and a deleting phase while learning the given input-output associations. The adding phase develops a crude structure by jilling in resources (connections, units and layers) at a virtual multilayer network with a maximum of L possible layers. The deleting phase then removes any unnecessary connections to obtain a rejned structure. The additions and deletions are done based on a sensitivi~ measure and a corresponding probability rule so that only the synapses which are most effective in reducing the output error are preserved. A generalization error estimated from a validation set is used to control the alternation between the two learning phases and the termination of learning. Simulations, including handwritten digit recognition, demonstrate that the algon”thmis effective in finding an appropriate network structure for a small network which can generalize well. The algorithm is used to investigate when the size of a network is importantfor generalization. O 1997 Elsevier Science Ltd.